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EVALUATION PROCESS: 
PROJECT SELECTION §& PROGRAM REVIEW 


1. INTRODUCTION 
1.1 This Canada/MAB report is the second dealing with the subject of 
project selection and program evaluation in the MAB context. The 
initial report* wa a first step toward developing a provisional 
framework of criteria to assess candidate proposals for designa- 
tion as MAB projects. In accordance with the strategy outlined 
in the first report, this document is intended to give guidance 
to all those interested in the process of proposal assessment 
and program evaluation. It is intended that this information 
will (1) assist a research proponent in writing up a research 
proposal for MAB designation; (2) be the base from which groups 
assigned the tasks of assessment, continuing review or over-all 
evaluation will work; and (3) provide (potential) funding agencies 
with explicit information on the criteria by which submitted propo- 
sals are judged relevant to MAB. To this extent it replaces 
Canada/MAB Report No. 1. 
1.2 In this report the following definitions are used: 
Proposal: those projects under consideration by the Joint MAB 
committees until the time they become designated 
MAB projects. 
Proponent: One (researcher, institution, centre, etc.) who 
submits a proposal to the Joint MAB committees 


for consideration for MAB designation. 


Provisional projects: projects receiving conditional funding, 
i.e., feasibility or seed funding. 


Project: a MAB-designated, fully operational piece of research. 


Evaluation: total review process consisting of assessment, period- 
ic review of progress during the life of each MAB 
project, postmortem and review when any project is 
finished, and the review of the total program itself. 


*Canada/MAB Report No. 1: Project Selection and Evaluation - Provisional 


Criteria, 3lst August, 1974. 


In order to more clearly distinguish between the two 
phases of the evaluation process, the following terms 
are used: 


Assessment: process by which a proposal becomes a 
MAB-designated project. 


Review: process by which a project is monitored once 
it has become fully operational. 


MAB Evaluation Panel: Group charged by CCMAB and ICMAB with the 
assessment of proposals and/or the review 
of MAB projects (one for each sub program). 
1.3 The evaluation process outlined herein is based on the present 
understanding of the Program objectives both national and inter- 
national* (for extracts see Appendices 1 and 2) and the means 
whereby these may be achieved in Canada. The evaluation process 
is considered an integral part of the MAB program and, therefore, 
as understanding about the program is increased, the process itself 


and the directions of the program, may change. 


1.4 It is to be clearly understood that proponents must address 

the criteria when submitting their detailed proposals (and by 
implication, in their Letters of Intent). Those proposals not 
addressing the criteria cannot be considered for MAB designation. 
It is for this reason that four sub-program co-ordinators have 
been added to the MAB secretariat to aid proponents in the de- 
velopment of their proposals and to act as an interface with 
research directors, the Joint MAB Committees and, when appro- 


priate, potential funding bodies. 


r Prospectus for Canada/MAB 1973. 


international coordinating Council of the Programme on Man and the Biosphere. 
First Session, Paris 9-19 November, 1971. 


2. THE PROCESS OF EVALUATION 

2.1 The evaluation process might be thought of as a series of sequent- 
ial steps, iinked vy feedback loops. It begins with the formula- 
tion of a research proposal, continues with an assessment of the 
merits of the latter, and a decision whether or not MAB design- 
ation is appropriate. Once operational, a MAB project is periodi- 
cally reviewed throughout its life. The evaluation process is 
not complete for any project until results are fully synthesized 


and insights fed back into the MAB process. 


2.2 Four or five groups of "actors" are involved: 
research proponents with MAB - designated projects; funding 
agencies for those MAB-designated projects requiring financial 
support; citizens who are interested or involved or who would 
be affected by the proposed research or its results; the 
decision-makers who would be -xpected to have direct interest 
in the outcome of the project; and CCMAB and ICMAB who are 


interested in the over-all outcome of the program. 
2.3 Figure 1 provides a schematic representation of the process. 


2.4 In order to stimulate preparation of proposals, information about 
the program must be available to potential researchers. Documents* 
have been prepared by CCMAB and ICMAB and circulated by the Secre- 
tariat (including Sub-Program Co-ordinators) and by the Joint MAB 


Committees. More information may be furnished through direct 


“Prospectus for Canada/MAB, 1973 
Canada/MAB Reports Nos. 1, 2, 3, 4. 1974. 
Communiqué, No. l. July 1974. 


contacts with potential researchers or by the holding of work- 
shops, seminars, etc. Those proposals generated would then follow 
through the three stages of an ongoing project and program 


evaluation process as well as with a final output or feedback stage. 


2 aS Stage iRe Proponent's Submission of a Letter of Intent 


2.5.1 After preliminary discussions with the MAB Secretariat, 
a research proponent would submit a Letter of Intent, 
outlining in broad terms the area of study, its object- 
ives, its components, personnel to be involved, insti- 
tutional arrangements, and the approximate level of fund- 
ing required and/or available. The MAB Secretariat, in 
cooperation with the appropriate MAB Evaluation Panel 
would assess such Letters of Intent for general consist- 
ency With the Criteria olitlined an this report, and the 
relevant sub-program framework and would decide whether 
the proponent should be encouraged to proceed further, 
or whether the proposal is likely to be inappropriate 
or inadequate in some way as a MAB project. Proponents 
may come from federal or provincial agencies, universities 
or thet private sector. 

2.5.2 Should the results of this preliminary assessment be 
positive, then the MAB Secretariat, if appropriate, 
would initiate discussions with potential funding agencies 
to obtain an expression of interest. At the same time the 
proponent would be required to contact local or provincial 


officials for expressions of support and interest (moral 
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or financial) in the proposal. A small grant to initiate 
a feasibility study or for building the core management 
(institutional) group may be considered at this time. 
In such a case the proposal would become a Provisional 
Project. 

2.5.3 Should the proposal be considered inappropriate for MAB 
but potentially relevant to some other research program 
or agency mission, it would be passed on to the appropriate 
agency for independent consideration. 

2.5.4 Letters of Intent from federal or provincial agencies could 
concern research about to be initiated or already under way. 
It is understood in this instance that this would constitute 
official interest, 

2.6 Stage 2: Proponent's Submission of a Formal Proposal 

2.6.1 The second stage would consist of a formal detailed submission 
from a proponent with help from the appropriate coordinator 
to the MAB Secretariat, and its subsequent assessment by the 
latter and the appropriate MAB Evaluation Panel. This stage 
would consist of two steps. 

2.6.2 The first step would be an initial review of the document- 
ation submitted. This would result in a decision to: 
a) proceed with the second step, 
b) refer the proposal back to the proponent for revision, 
c) reject the proposal as inappropriate to the MAB Program. 

2.6.3 When a proposal seems to be clearly within the perspective 


of the Canadian MAB Program, the second step of Stage 2 would 


2eOit 


2 
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4 
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be to conduct an on-site review by representatives of 
the Joint MAB Committees, decision-makers, potential 
funding agencies, and/or others as appropriate. Here 
the aim would be to explore in greater depth with the 
proponent the various aspects of the proposal and to 
examine the available facilities, to meet core staff 
associated with the proposal, senior administrators 

of the research agency whose support would be required 
for successful pursuit of the study and other agencies 
adentifiredfasatanterestedapartiesys 

Based on the report of the on-site review and recommend- 
ation of the Evaluation Panel, the Joint MAB Committees 
would then make the decision to designate or reject the 
proposal as a MAB project. .If the proposal receives 
designation, and financi~1 support is not already avail- 
able, representation on behalf of the MAB project wouid 
be made to the appropriate funding agenciés. 
Immediately following MAP designation of a project, 
information appropriate to that project would be sent 
to Paris for incorporation into the international MAB 
Program and to other national committees potentially 
interested,in thateprojectsr) Itvas possable that some 
Canadian projects will be invited to become part of the 
International Core Program but details of this aspect 


are not yet available. 


yAn sdf Stage 3: Review of Progress 
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The third stage would consist of the monitoring of progress 
of the research, involving the submission of periodic reports 
and occasional site visits arranged by the MAB Secretariat 
and/or the support agency. Where appropriate, suggestions 
would be offered as to ways in which the objectives of the 
research might be more effectively pursued. The MAB Eval- 
uation Panels will have the over-all responsibility for 
evaluating the various studies undertaken under the auspices 
of the Canadian MAB Program, and for ensuring that required 
syntheses are provided, while the MAB Secretariat, subject 
to approval by the Joint MAB Committees, would ensure that 


appropriate reports are made to Paris when required. 


2.8 Final Output and Feedback Stage 
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In keeping with the MAR concept, all the "actors" will be 
involved in the interactive process throughout the life 

of a MAB project. Emphasis on this interaction mode is 
an important characteristic of the MAB process. It has a 
number of fairly obvious advantages in terms of inform- 
ation transfer and enhances the traditional "product 
orientation" of research. 

The links between researchers concerned with a particular 
problem area, will, through the MAB Program, extend beyond 
the direct researcher - researcher contacts to emphasize 
transfer of information to decision-makers and other cons- 
tituents in a way to lead to more rational management of the 


earth's resources. 


2.8.3 Thus the outputs of a MAB project will be to provide data 
for the ongoing evaluation of the over-all goals and 
achievements of the MAB Program nationally and international- 
ly. The conclusions and recommendations of such evaluations 
will in turn be fed into the formulation of future Canadian 
MAB goals and programs, as indicated in Figure 1, and into 


the international system. 


3. CRITERIA FOR EVALUATION 

3.1 In the development of a process of evaluation, the establishment of 
a set of criteria is critical not only for the assessment of proposals 
but for their continuing review during their life as MAB projects. 
Decisions as to the appropriateness and the relative merits of 
proposals submitted for designation as MAB projects should be 
based upon logical and broadly accepted criteria that are clearly 
specified to proponents. Two sets of criteria, originally described 


in Canada/MAB Report 1, are involved: Contextual and Generic. 


3.2 Contextual criteria: Contextuai criteria consist of considerations 
which are considered to be pre-conditions for project acceptance 
within the terms of reference of the international and Canadian 
MAB programs. For any proposal to receive designation by the 
Canada/MAB Committees, it would need to: 

(a) be within the competence and responsibilities of the UNESCO 
Man and the Biosphere Program; 
(b) avoid duplication of work being undertaken by, or under the 


auspices of other international organizations; 


=GJ 


(c) be consistent with the objectives of the Canadian MAB Program 


as set out in the Prospectus for Canada/MAB and subsequent 


Canada/MAB reports. 


3.3 Generic Criteria: Generic criteria reflect specific concerns 
about the scientific, societal, and practical aspects of the 


proposals. 


3.4 These generic criteria take on particular significance in the 
case of the MAB program because of its interdisciplinary 
nature, and its focus upon societal problems arising from the 
use or allocation of natural resources. This type of research 
is relatively new, and there is a growing recognition that 
existing methodologies are unsatisfactory for the assessment of 
interdisciplinary research proposals and for their over-all 


evaluation. 


3.5 In interdisciplinary research, the objective pursued extends 
beyond the satisfactory solution to a precisely defined problem. 
In fact, it is here that the dilemma lies, as the problem to be 
solved is usually ill-defined, a cluster at interrelated sub- 
problems, and is often in the terminology of Chevalier,* a 
"meta-problem". Its solution may not be achievable and, in fact, 
the problem itself may change with the passage of time and change of 
perception. The necessity for flexibility and for keeping options 


open is therefore critical. 


* Chevalier, Michel. Social Science and Water Management: A planning Strategy. 


Policy and Planning Branch, Department of Energy, Mines 4 Resources, Ottawa, 
Canada, 1969. 
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3.6 Thus the orientation, although directed toward the development 


os 


if 


of new knowledge is also directed towards the synthesis of know- 
ledge from widely differing disciplines in order to provide more 
useful information as a basis for policy and/or management decisions. 
That is, the problem is defined first in societal terms and thus 
various pertinent bodies of knowledge are called upon for solution 
to the problem allowing the final results to be synthesized and 
expressed in response to the defined societal (human) objectives. 

It also follows that every effort must be made by the researcher 

to become informed of similar studies that have been done and the 


relevancy of their results within the MAB context. 


While few precedents exist for either the conduct or the evaluation 
of interdisciplinary research, experience to date points to several 
key requirements. Scientific rigour and thoroughness are prime re- 
quirements at all levels - coucept development for the over-all 
study, individual research contributions, synthesis, and testing 

of the social hypothesis which provided the original motivation. 
Scientific excellence in an interdisciplinary program will frequent- 
ly mean that old assumptions, or those inherent in the current under- 
standing, will need to be questioned and re-evaluated. Frequently, 
the assumptions held by one disciplinary group will conflict with 
those of another. Thus confidence in the competence of team 

members and a trust in their integrity is vital in the development 
of an appropriate managerial organization for an interdisciplinary 
research team. This implies an explicit commitment on the part of 


each researcher to plan his part of the project with goals in mind 


sa 


= 


which transcend his discipl: nary interests and contribute to the 
total project. Of course, time is essential for confidence and 
trust to develop. It also implies a commitment on the part of 


the institution where the work is being done. 


An important and continuing part of the research process is one 


of concensus-building in developing a dynamic framework for 


co-ordination of the project designed to systematically test one 


assumption or input against other related inputs. This process 
also must be rigorous. The concensus-building process also 
serves as a learning process whereby actors - researchers and 
users alike - can gain a better understanding of the issue under 


study and the research objectives. 


Potential users, or those which could be affected by application 


of research findings, should *e considered as partners in the 


research and the associated learning process. Perhaps this is 


the most significant departure from traditional research pursuits 
and the most difficult to accomplish. However, human needs 


cannot be perceived by a research team working in isolation, 


and strategies for user involvement, review, information 


dissemination and feedback are considered a requisite part of 


all MAB-designated projects. 

To be more specific and to summarize what has been said, the 

statements above, may be subdivided into two broad aspects: 
(a) MAB Process; 


(b) Research Considerations. 
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These aspects are further defined by the following specific 
statements (criteria) which must be addressed by proponents 


(no order of priority implied): 


MAB Process: 


- Contribution applicable to the understanding of the MAB 
problématique 


- Implications for or potential contributions to public 
policy strategies 


- Evidence of interdisciplinary methodology and commitment 
to interdisciplinary principles 


- Provision of managerial organization appropriate to 
operation and continuing review of interdisciplinary 
research 


- Provision of a dynamic framework co-ordination with 
relevant policy agencies 


- Mechanisms for communications with other constituent 
groups 


- Provision of public/popular information 


- Contribution to the techniques of analysis or systhesis 


Research Considerations 


- Provision of formal training to specialist groups 
and public within the research program 


- Evidence of integration and extension of previous 
studies 


- Availability and commitment of personnel and facilities 
- Contribution relative to the State of the Art 
- Competence of researchers 


- Scientific rigour and thoroughness included in both 
design and method 


- Contribution to the definition and understanding of 
a societal problem 


-13- 
4. THE PROCESS OF CONTINUING REVIEW AND OVER-ALL EVALUATION 
4.1 Although of considerable importance to the denouement of a 
MAB project and to the over-all evaluation of the MAB program, 
the precise stops of continuing review and over-all evaluation 
have not yet been defined. It is here that the first MAB projects 


will play an essential role in the development of MAB in Canada. 


4.2 Since there is a scarcity of postmortems on any of the inter- 
disciplinary studies carried out to date*, it is exceedingly 
difficult to set down exactly what steps should be followed and 
what problems should be foreseen, allowed for and/or avoided. 
However, specific attention should be paid by the proponents to 
the question of how problems will be recognized and the organi- 


zational procedures for dealing with them. 


4.3 In the continuing review, it is expected that each project director 
will set suitable schedules f>r work to be completed, results 
synthesized and reports prepared. However, the efficacy of these 
schedules will be reviewed by the appropriate MAB Evaluation Panel. 
It is understood that changes ‘n direction may result from new 
knowledge being obtained and that. some sub-projects may have to be 
added or subtracted from the original MAB project design. It is 


here that the MAB Evaluation Panels can be most useful. 


i 


“for example B.W. Mar, Problems encountered in multidisciplinary resources 
and environmental simulation models development. J. Environmental 
Management 2¢ 85-100 (1974). 
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4.4 


4.5 


4.6 


Sin 


Each MAB Evaluation Panel will meet at least twice per year 

but more Often if deemed necessary. Each one will prepare an 
annual report for ICCMAB and CCMAB on the status and progress 

of each MAB project under their care with any recommendations for 
action deemed sinraserare® 

The complete secretariat together with the chairman of each of 
the MAB Evaluation Panels and such others as the Chairman of 
ICMAB and CCMAB select will act as a general integrating and syn- 
thesizing committee for the et Canada/MAB program. They shall 
be responsible for over-all coordination and synthesis, identifi- 
cation of overlap, and direction of the results of one study to 
another where they may be useful. 

This group will also be responsible for the over-all evaluation 
of the program and will recommend to ICMAB and CCMAB such changes 
in the program as are shown to he necessary as a result of their 
evaluation. As with the MAB Evaluation Panels, feedback into the 


system is expected. 


SOME PRACTICAL PROBLEMS IN ASSESSING PROPOSALS 


=i 
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Even though a set of criteria has been identified for the evalua- 
tion of the MAB research, there remain a number of practical 


difficulties in assessing the merits of particular proposals, and 


in comparing a given proposal with others competing for designation 


within the same sub-program. Among these practical problems are 


the following: 


~15- 


(a) some attributes of research proposals are difficult 
to express in precise terms, e.g., the economic 
and social significance of the development or refine- 
ment of theory which might result from a given project. 


(b) performance with respect to one set of criteria might be 
measurable in one kind of unit, and performance with 
respect to others in different kinds of units. Although 
a broad range of attributes are measurable directly 
or indirectly in monetary terms, many are not, e.g., 
the value of the discovery that a certain key species 
or group of species is endangered. 


(c) different importance may be attached to the various 
criteria that are applied, and means need to be devised 
to take this into account, e.g., it may be considered 
more important that research in a particular area will 
lead to a practical solution to a chronic and worsen- 
ing problem (such as the erosion of beaches, or disposal 
of industrial wastes) which can provide precedents for 
Similar approaches in other areas than to the development 
and/or testing of theory for future use in policy/manage- 
ment decisions. 


(d) the long-range versus short-term nature of a problem; 
for examole a) in an area where there are conflicting 
pressures for land utilization, research results are 
likely to have an immediate impact on decisions. 
While it is clear -nat information is vital before 
pressure groups or inaction result in a loss of options, 
it is also clear that long-term implications may be under- 
played or ignored; b) with a study on energy utilization 
(increasing, decreasing, static), where the long-range 
impact cannot be predicted and the only assessments of 
possible impacts a:e« provided by "people's" options, yet 
the influence of the project on "people's" thinking 
could be highly significant in the short to medium as 
well as long term. 


5.2 Such problems have been discussed by various authorities on the 
evaluation of research as well as by scholars concerned with the 
allocation of capital and labour among competing alternative uses, 
including research (Dortman, 1965*). These observers have been criti- 


cal of the use of certain tools of evaluation (such as benefit-cost 


analysis), and especially attempts to express all attributes in 


I RR SDAPT ATE ye Sa 2 Se one PS PE Pe ee ee 
* Dortman, R. (ed) Measuring Benefits of Government Investments, Washington, D.C.: 
Brookings Institution, 1965. 


nies 


monetary terms. They have been particularly skeptical of the 
application of such methods to the evaluation of projects or 
policies involving the consideration of "intangibles", such as 
aesthetics, environmental iisruption, or effects of life styles. 
Several observers have commented upon the misuse of these tech- 
niques in the appraisal of research projects (Crutchfield, 

Kates and Sewell, 1967*; and Scherer, 1965**) and have called for 


new approaches. 


5.3 A number of alternative approaches to evaluation have been suggest- 
ed in recent years. Among them, that described by J.C. Holmes*** 
is particularly attractive in the MAB context because it presents 
an orderly method for Seanudt Tag a wide range of subjective opinions. 
In a sense, it attempts to reproduce in an explicit way many of the 
sub-conscious steps involved in the process through which individuals 
go when making an important decision. The method is clearly not a 
calculation in the accepted sense of the word, rather it is a way 
of presenting a complex argument systematically. Of particular bene- 
fit is the fact that assessors develop a much deeper understanding 
of the criteria by use of this method. What follows has been large- 


ly taken from the Holmes paper. 


5.4 Underlying the process are a number of basic assumptions: 


* Crutchfield, J.A., R.W. Kates and W.R.D. Sewell. Benefit-Cost Anlaysis 
and the National Oceanographic Program. Natural Resources Journal 
7 (3) 5° 3615 3752719672 


** Scherer, F.M. Government Research and Development Programs, in Dortman, R. 


(ed) Measuring Benefits of Governments, Washington, D.C.:Brooking 
Institution, 1965. 


*** Holmes, J.C. An Ordinal Method of Evaluation, Urban Studies 9 (1): 
179-191, 1972. 


a) 


b) 


Cc) 


qd) 
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that arithmetical operations are inappropriate for 
dealing with unquantifizble data or criteria (i.e., 
under conditions where there is no absolute scale), 
or it is questionable tc use conventional measures 
such as .onetary values; 


that while some planning criteria are clearly more 
important than others, it is not realistic to 
assign relative numeric values to the relationship 


that proposals can be rated against each criterion 
in terms of their relative performance, even when 
there is no absolute scale; 


that the application of an ordinal method such as 
this makes visible the relative significance that 
has been awarded to the various factors affecting 
the decision. 


The process itself involves a number of sequential steps: 


a) 


b) 


Cc) 


d) 


e) 


selecting the criteria and, if appropriate, their 
groupings; 


determining the relative "classes of importance" 
of the criteria, and their groupings; 


awarding relative levels of performance for each 
proposal against each criterion, and displaying 
in a. matrix? 


identifying "corresponding positions" by diagonal 
shifting of subsequent rows of the matrix one 
column to the right; 


analyzing the resu] ant diagonal matrix to determine 
the preferred order. 


5.6 In carrying out this process certain "rules" have been establish- 


ed, as follows: 


a) 


b) 


disqualification against one important criterion 
implies total disqualification; 


it is not necessary to make unique performance assess- 
ment - two (or more) proposals can be judged to 
perform equally well and therefore be awarded a "tie" 
rating; 


=t8= 

c) in general, significant decisions using this method 
require that the number of "classes of importance" 
for the criteria be less than the number of proposals 
to be evaluated; the essential difference between 
a higher and a lower class of importance is that the 
lower class of importance should only influence the 
decision when there is a tie at the higher class; 


d) the proposal with the higher number of first "positions" 
will be first; 


e) no number of lower ''positions' can be given precedence 
over, or equivalence with, a higher position. 


Seo, oteshowlds be pointed out immediately, however, that the method aS 


not intended to be definitive, but rather an aid to the decision- 

makers. The final decision may rest not only on the results of 

this comparison but also on other issues not known to the assessor(s); 
perhaps on political, economic, or even on measurable criteria which, for 
reasons outside the assessor's mandate have to be separately deter- 
mined. Thus, for example, if the ordinal method of ranking gives 

an overwhelming number of second places to one proposal, it is 

obviously open to the final decision-makers to decide this consisten- 

cy is of more total importance than one or two first places by 

competing proposals against particular Crrceria, 1.6. contrary £0 
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5.8 The Holmes method of evaluation seems to be especially appropriate 
for the assessment of MAB-related proposals. There are several 
criteria that must be taken into account but they are not of equal 
importance. Effects of outeoee cannot always be quantified in mone- 
tary terms or in terms of conventional units, thereby requiring 
qualitative assessments. There are certain "minimum" criteria that 


must be met before a proposal can be considered for relative ranking. 


oao 
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All proposals for MAB designation must meet the Contextual 
Criteria noted above before they can be considered any further. 
The Generic Criteria grouped into "MAB Process" aspects 

and ''Research Considerations" serve to determine the relative 
merits of the various competing proposals within each sub- 


program area. 


In order to use the Holmes' method it is first necessary to 
determine the relative "class of importance" or rank for each 

of the criteria within the two major groups (aspects) and the 
relative class of importance of the groups themselves. Although 
Holmes emphasizes the importance of this step, he does not provide 
much guidance. The Joint MAB Committees considered that a logical 
and systematic method of making this determination would be to 

use what might be termed a “pairs analysis" technique. This 
technique permits the systematic evaluation of every pair combi- 
nation, in which the subjective evaluation of the relative merits 
(better than or equivalent to) is recorded. It is a fully participa- 
tive process which can be "democratized" to the extent of recording 


positive and negative views, not simply majority views. 


In order to assess proposals it is necessary to examine each one 
relative to other competing proposals and ranking them in order 


of "performance" with respect to each criterion. A matrix 


illustrating this ranking result for proposals A, B, C and D might then 


be constructed and look like Table l. 
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HYPOTHETICAL MATRIX ILLUSTRATING "CORRESPONDING 
POSITIONS' FOR EVALUATION OF PROPOSALS AGAINST 
SELECTED CRITERIA 


ee ee er nae yeaa ea eT SE, |, en, En LL eee a en et 


Performance 
Criterion Class of Importance Ist 2nd 3rd 4th 


Le alpha I A B C D 
ran beta ug D A B C 
Os gamma Il B D A C 
4, delta III C A D B 


5.11 Remembering that the matrix shows qualitative relationships both 
horizontally and vertically, it is clear that proposal A has been 
judged the best "performer'' for the most important criterion. In 
so far as B is concerned, it is second best performer for criterion 
1 and best performer for criterion 3 (which has the same class of 
importance as criterion 2), but D is the best performer with respect 
to criterion 2; thus both B and D follow A but it is not possible 
to immediately determine which of these two is closer to A. Ina 
Similar way, it can be seen that C is the third best performer with 
respect to the most important criteria, and A is the second best 
performer with respect to criterion 2 and D is the second best 
performer for criterion 3. These are said to be in "corresponding 
positions", The entire "corresponding position" matrix can be more 
easily understood by reconstructing the table by shifting successive 


rows one column to the right as in Table 2. 


ae 


TABLE 2 


HYPOTHETICAL MATRIX ILLUSTRATING "CORRESPONDING POSITIONS" 
FOR EVALUATION OF PROPOSALS AGAINST SELECTED CRITERIA 


Class of Corresponding Positions 
Criteria Importance Ist 2nd 3rd 4th Sth 6th 7th 
alpha I A B C D 
beta Lt D A B C 
gamma II B D A C 
delta Lit C A D B 


An analysis of this matrix to summarize the distribution of the 


proposals according to their "corresponding positions" shows: 


Corresponding Positions 
lst 2nd 3rd 4th Sth 6th 


A 1 0 1 Z 0 0 
B 0 2 0 1 0 1 
C 0 0 2 0 2 0 
D 0 i! 1 1 1 0 


According to the rules, the preferred order or ranking would 


be A; Boo e: 


5.13 It is hoped that this hypothetical example will not only serve 
as a practical guide, but also as an indication of the care required 
in establishing the Class of Importance of the criteria and the 


relative performance of the various proposals. Of course in the 
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application of the method in relationship to each sub-program area 
of MAB, the relative importance of the criteria need not be perceived 
to be the same. This does not in any way invalidate the process but 


rather emphasizes its flexibility.. 


6. CONCLUSION 

6.1 This report sets out the process of evaluation by which the achievement 
of the MAB program can be monitored. The process is continuous, but 
for the sake of ease in distinguishing the phases of the process, 
two aspects are also described. The process by which a proposal 
becomes a MAB project - assessment; and the process by which a project 
is monitored during its life - review. 

6.2 Criteria which are to be used in this process are set out along 
with the method by which they are to be used. 

6.3 The Canada/MAB Report 1: Project Selection and Evaluation - 
Provisional Criteria is thus replaced by this report. 

6.4 Thus this report is a result of the dialogue initiated between 
Canada/MAB and its potential constituents called for in the Canada/ 
MAB Report 1. It will be used for the assessment of all proposals 
received by the CCMAB and ICMAB Joint MAB Committees and those 
proposals not addressing the criteria will not be further considered. 

6.5 It is hoped this document will fill the very important need of 
outlining to the MAB constituents the types of proposals which will 


be considered within this program. 
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For further information please address all correspondence to: 


Patricia Roberts-Pichette, Ph.D. 
Executive Secretary 

Canadian MAB Program Secretariat 
Liaison and Coordination Directorate 
Environment Canada 

Ottawa, Ontario. 
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ANNEX 1 


EXTRACTS FROM INTERNATIONAL CO-ORDINATING COUNCIL OF THE PROGRAM ON MAN 


AND THE BIOSPHERE. FIRST SESSION. PARIS 19-29, NOVEMBER, 1971.* 


1¢ & 


by 


SCOPE AND OBJECTIVES OF THE PROGRAMME 


General Scope of the Programme 


The Council recalled that the Man and the Biosphere Programme is an 
interdisciplinary programme of research which emphasizes an ecological 
approach to the study of interrelationships between man and the 
environment. It will be implemented in close co-operation with the 
organizations of the United Nations concerned and the competent inter- 
national non-governmental organizations. It will focus on the general 
study of the structure and functioning of the biosphere and its ecologic- 
al divisions, on the systematic observation of, and research on, the 
changes brought about by man in the biosphere and its resources, on the 
overall effects of these changes upon the human species itself, and on 
the education and information to be provided on these matters. 


The very nature of the Programme gives it necessarily a wide scope. 
Generally speaking, it will be concerned with subjects of global or 

major regional significance rather than problems of local importance 

that can best be handled at the national level and that do not speci- 
fically require international co-operation. Being intergovernmental, 

it will concentrate on activities where governmental intervention or support 
is a condition for success. 


Interdisciplinary studies on selected environmental problems will form 
central and essential components o. the Programme. Other important 
problems will be only tangential to the Programme, in part because they 
are already covered by other international programmes, with which close 
liaison will be arranged. Thus, although the programme is entitled 
"Man and the Biosphere'', it will not be directly concerned with the 
oceans. In so far as the phenomena which occur there are directly 
related to terrestrial problems, arrangements will be made with the 
Intergovernmental Oceanographic Commission (10C), which is the focal 
point for development and co-ordination of the Long-Term and Expanded 
Programme of Ocean Exploration and Research (LEPOR). Similarly, projects 
dealing principally or exclusively with hydrological or meteorological 
problems will be included, since they are already handied within the 
framework of the International Hydrological Programme and relevant WMO 
programmes. Here again the necessary links will be established with 
these scientific programmes whenever required. Problems of a strict 


* includes amendments of council meetings in 1973, 1974. 
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management nature, related more specifically to agriculture, industry 
and health, which are’ receiving major attention from various inter- 
governmental agencies, such as FAO, UNIDO and WHO, and which are 
connected with problems of choice and decision of a non-scientific 
character, will not be included under the Programme. Similarly, manage- 
ment problems of pollution control and of urban and rural development 
are not included. The objective of most research themes and projects 
under the Programme will rather be to obtain the scientific information 
required to facilitate the solution of these problems. 


Though at the present time no definite duration for the Programme has 

been set, it is thought that the main objectives of a number of projects 
can be realized satisfactorily within a period of less than ten years. 

It is considered important, therefore, to make provision for participating 
countries to undertake periodic and critical evaluation of projects, 
appraisal of progress achieved and consideration of future action to be 
taken. 


Objectives of the Programme 


The Council, having reviewed the objectives set forth in document 
16 C/78 decided to formulate the following objectives for the Programme: 


The general objectives of the Programme is to develop the basis within 

the natural and social sciences for the rational use and conservation 

of the resources of the biosphere and for the improvement of the global 
relationship between man and the environment; to predict the consequences 
of today's actions on tomorrow's world and thereby to increase man's 
ability to manage efficiently the natural resources of the biosphere. 


With this general objective in mind, the Programme is intended more 
specifically to develop a limited number of projects: 


(1) to identify and assess the changes in the biosphere resulting 
from man's activities and the effects of these changes on man; 


(2) to study and compare the structure, functioning and dynamics 
of natural, modified and managed ccosystems; 


(3) to study and compare the dynamic interrelationships between 
"natural" ecosystems and socioeconomic processes, and 
especially the impact of changes in human populations, 
settlement patterns and technology on the future viability 
of these systems; 


(4) to develop ways and means to measure quantitative and 
qualitative changes in the environment in order to establish 
scientific criteria to serve as a basis for rational manage- 
ment of natural resources, including the protection of nature, 
and for establishment of standards of environmental quality; 
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(5) to help bring about greater global coherence of environmental 
research, by; 


(a) establishing comparable, compatible and , where 
appropriate, standardized methods for the 
acquisition and processing of environmental data; 


(b) promoting the exchange and transfer of knowledge on 
environmental problems; 


(6) to promote the development and application of simulation and other 
techniques for prediction, as tools for environmental management; 


(7) to promote environmental education in its broadest sense, by: 


(a) developing background material, including books and 
teaching aids, for educational curricula at all levels; 


(b) promoting specialist training in appropriate disciplines; 


(c) stressing the interdisciplinary nature of environmental 
problems ; 


(d) stimulating global awareness of environmental problems 
through public and other information media; 


(e) promoting the idea of man's personal fulfilment in 
partnership with nature, and his responsibility for 
nature. 


III. CRITERIA FOR SELECTION OF PROJECTS 


The Council recommended the following as essential criteria for the 
selection of projects for the Man and the Biosphere Programme: 


es) That the project would provide information, through research on 
natural and social sciences (including survey or repeated survey), 
essential to rational decisivn-making about the use of natural 
resources; 


(2) That the necessary research is feasible and likely to produce 
results, in the short- and medium-term, of sufficient 
precision for the use that is to be made of them; 


(3) That significant progress would be enhanced by international 
co-operation. Through co-ordinated planning and execution, 
the use of compatible or standardized methods, and the avail- 
ability, interchange and synthesis of information; 
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(4) That the project lies within the field of competence and 
responsibility of Unesco though it may contain some elements 
that are within the competence of other intergovernmental and 
non-governnental organizations; 


(5) That it should have intrinsic merit as a critical programme of 
research which will advance knowledge having a bearing on the 
interrelatianship between man and the biosphere. 


The Council further recommend as criteria for priorities: 


(a) 


(b) 


(c) 


(d) 


(e) 


(£) 


That it should be interdisciplinary, * either in terms 

of scientific disciplines. or.an the, sense that it 
includes studies of interactions between human populations 
and the biosphere; 


That it should be of direct assistance and economic 
Significance to developing countries; 


That it should be based, where possible on certain 
selected centres, where appropriate facilities are 
already available and related research is in progress 
which could be economically developed; 


That it could advantageously be linked with a programme 
of training, especially of ecologists with some knowledge 
of the social sciences or vice versa; 


That it should lend itself to use for education, 
demonstration and extension work; 


That it should produce significant progress in solving 
a problem through support provided by the Programme. 


SCIENTIFIC CONTENT OF THE PROGRAMME 


Project No. 1 


PYOJECE NOs 2 


PYOVECEUNO car 


Ecological effects of increasing human activities 
on tropical and subtropical forest ecosystems. 


Ecological effects of different land uses and 
management practices on temperate and mediterranean 
forest landscapes. 


Impact of human activities and land use practices 
on grazing lands: savanna, grassland (from temperate 
to arid areas), tundra. 


Subsequent meetings of ICC-MAB have stressed this aspect and of placed 
emphasis on the importance of the note of the social sciences in the 
Programme and the necessity for the integration of their expertise in 
all aspects of planning. 


Project No. 4 Impact of human activities on the dynamics of 
arid and semi-arid zones' ecosystems, with particular 
attention to the effects of irrigation. 

Project No. 5 Ecological effects of human activities on the value 
and resources of lakes, marshes, rivers, deltas, 
estuaries and coastal zones. 

Project No. 6 Impact of human activities on mountain ecosystems. 


Project No. 7 Ecology and rational use of island ecosystems. 


Project No. 8 Conservation of natural areas and of the genetic 
material they contain. 


Project No. 9 Ecological assessment of pest management and 
fertilizer use on terrestrial and aquatic eco- 
systems. 


Project No. 10 Effects on man and his environment of major 
engineering works. 


Project No. 11 Ecological aspects of urban systems with particular 
emphasis on energy obligation. 


Project No. 12 Interactions between environmental transformations 
and genetic and demographic changes. 


Project No. 13 Perception of environmental quality 


Project No. 14 Research on environmental pollution and its 
effects on the biosphere. 
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ANNEX 2 


EXTRACT FROM PROSPECTUS FOR CANADA/MAB * 


Si. PROGRAM DEFINITION 


3.1 The potential scope of the MAB program and the range of activities 
that might be undertaken as components of it are almost unlimited. 
A preliminary survey by CCMAB/ICMAB has revealed a large number and 
wide variety of research activities in Canada that could be related 
in on2 way or another to the broadly stated international project 
objectives for MAB. It is necessary, therefore, to establish 
guidelines for Canada/MAB and these must be in harmony with the 
pattern of Canadian priorities. 


3.2 The international time scale for MAB is loosely defined,** It is 
proposed that the Canada/MAB program should extend for at least 
five years but not extend ten years. 


3.3 The following guidelines for Canada/MAB have been developed as a 
result of extensive consultation among the members of CCMAB and 
ICMAB. The guidelines that have been defined related to both 
objectives and procedure and are of different levels of generality. 
They are, therefore, set forth under four readings, as follows. 


3.31 General objectives - an interpretation of the 
internationally approved objectives** in the light 
of Canadian requirements. 


3.32 Characteristics - an interpretation of international- 
ly approved criteria in the light of Canadian requirements 
with the additional statement of program characteristics 
demanded by Canadian conditions. 


3.33 Research themes - a statement of problems of bread 
social and environmental significance, the solution 
of which can be sought directly through large co- 
ordinated research undertakings and contributed to 
by the work of various individuals and institutions. 


3.34 Canadian Social/Environmental Problems - problems 
with potential regional definition within which 
research themes can be caught up, and in relation 
to which specific projects and project objectives 
should be developed. 


* As modified by later Canada/MAB Reports 
** International Co-ordinating Council of the Program on Man and the 
Biosphere. First Session, November 1971. 
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3.4 GENERAL OBJECTIVES 


3.41 To provide information which will as quickly as 


Gs4e 


a c, 


possible increase understanding of man's inter- 
action with his environment and assist society to: 


(a) assess the effects of and control changes in 
the biosphere arising from man's activities; 


(b) increase or maintain resource yields while 
reducing adverse environmental affects and 
increase understanding of the various alter- 
natives and their consequences ; 


(c) enhance social opportunities. 

To enhance, through education in the general sense, 
the insight of individuals into man/environment 
problems and their alternatives solutions and to 
provide, through training, an adequate number of 
persons able to deal effectively with man/environment 
problems. 


To assist in the formulation of policy alternatives 
with respect to: 


(a) the protection of the environment; 
(b) the utilization of resources; 
(c) economic development; 


(d) enhancement of tie quality of life. 


3.5 CHARACTERISTICS OF THE MAB PROGRAM IN CANADA 


ey 


The distinctive and essential characteristic of MAB 
will be its comprehensiveness, attained by trans- 
disciplinary planning and execution at all levels. 
Program management must ensure close-co-ordination 
within and between natural and social scientists, 
wherever appropriate, at all stages of work and at 
all working levels. This is vital if objectives 

are to be met. In addition, the program is expected 
to stress innovation and integration rather than the 
incremental addition of isolated bits of information. 
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3.52 Planning and managing effective, integrated projects 
will be facilitated if the following characteristics are 
recognized as guidelines for planning. Canada/MAB 
project should: 


(a) start with a synthesis of existing data (i.e., 
development of a comprehensive project information 
base) ; 


(b) promote the application of simulation and other 
predictive technique based on systems analysis; 


(c) promote the experimental approach to ecosystem 
management ; 


(d) be based as far as possible in centres of 
opportunity, i.e. centres where the requisite 
research facilities already exist, and where 
spurs to integration would produce initiatives 
appropriate to the MAB program; 


(e) be of a scale large enough to be visible and to 
produce practical results, but small enough to 
be manageable and comprehensive; CONVERSELY 
projects should NOT be so ambitious as to appear 
impossible to conclude, nor so small as to yield 
insignificant results. 


3.53 Other distinctive characteristics of the MAB program are 
its orientation towards problem-solving and its emphasis 
on training, education and the dissemination of informa- 
tion. To retain that oriontation and those emphases in 
project selection, Canada/MAB projects should: 


(a) focus on problems of high public priority requiring 
solution within ten years, and, hopefully, contribute 
to the definition and solution of problems having 
social, economic and environmental dimensions that 
are as yet dimly seen; CONVERSELY, projects should 
NOT be directed toward problems that are primarily of 

academic interests; 


(b) produce usable results within five years; 


(c) produce results which can be related to effective 
planning for public policy development and decision- 
making; CONVERSELY, projects should NOT be conceived 
as cosmetics for predetermined development projects; 


(d) seek public participation. and provide for demonstration 
and communication to the public of project information; 


(e) incorporate plans for environmental education and the 
provision of training at the technical and profession- 
al levels; 


3.54 
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(f) employ a high proportion of young people at all 
levels and in all stages of the program. 


Since many programs of federal and provincial departments 
are already oriented to the solution of priority problems, 
the major tasks for MAB achieving the trans-disciplinary 
approach already outlined will be to provide in a few 
chosen areas, further dimensions of co-ordination and to 
arrange for sub-projects to fill gaps and round out 
projects so that they are more comprehensive and more 
readily integrated. It follows, then, that many MAB 
activities need not be costly in terms of ''new'' money. 


3.6 RESEARCH THEMES FOR CANADA/MAB 


3.61 


3.62 


3.63 


The ICC-MAB has proposed 13 projects, or research themes, 
seven related to geographically definable areas, five to 
problems arising from various general classes of human 
activity, and one to the difficult question of perception 
of environmental quality. These are all overlapping themes 
and do not really constrain our determination of the 
Canada/MAB program. 


After extensive study and discussion by social and natural 
scientists throughout Canada,. the following major themes 
were identified as of major significance to Canada, and 
found to be in harmony with the international guidelines: 


(a) social and © ivironmental causes and consequences 
of current trends in energy supply and demand; 


(b) social and environmental causes and consequences 
of alternative land-use practices related to the 
supply of forest products, food and recreational 
opportunity; 


(c) human perception of environmental change, including 
its relationship to education, the availability of 
information and other aspects of the social environment. 


It is obvious that these themes should be broadly inter- 
preted. Thus the extent to which each might be reflected 
within a project could vary depending on the specific 
project objectives. 


3.7 CANADIAN SOCIAL/ENVIRONMENTAL PROBLEMS 


The Canadian social/environmental problems considered most appropriate 
for the development of sub-programs of Canada/MAB are: 


3.71 how to achieve a desirable pattern of human settlement, 
land use and environmental quality in Canada in the 
circumstances of changing population, urbanization and 
industrialization. 


3.72 how to achieve maximum benefits and avoid adverse 
effects on human social structures, wildlife, soil 
and water quality, in an age of increasingly 
intensive agricultural and forestry management 
practices (including fertilizer and pesticide use 
and waste disposal). 


3.73 how to achieve maximum benefits and avoid adverse 
effects on human social structures, biological 
productivity, recreational opportunity and economic 
returns in the management of coastal ecosystems, 
particularly the Gulf of St. Lawrence and the Strait 
of Georgia; 


3.74 how to achieve maximum benefits and avoid adverse 
effects on social and cultural patterns, environment - 
al quality and wildlife productivity, in the circum- 
stances of technological change and industrial develop- 
ment in the Arctic, sub-Arctic and other isolated 
regions. 


Problems of these kinds recur in many parts of the world. The Canada/MAB 
experience will thus have wide international applicability. Particular 
studies abroad may be similar in concept to Canadian projects and 
Canada/MAB should encourage appropriate exchanges at all project stages. 


Although these problems as described above cannot be assigned directly 
to any of the ICC-MAB 1 projects, they each represent an amalgam of 
several of the projects. It is expected that many of the 13 projects 
will be more easily identified within the Canadian sub-programs as they 
are developed. 
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ANNEX 3 


MEMBERS OF THE MAB SUB COMMITTEE ON CRITERIA 


P.M. BIRD, (Chairman), International Program Branch, Liaison and 
Coordination Directorate, Environment Canada, Ottawa, Ontario 
K1A OH3. 


H.F. FLETCHER, Fisheries Research Board Service, 580 Booth Street, 
Suite 616, Ottawa, Ontario. 


P,. JACOBS, Faculté d'Aménagement, Université de Montréal, Montréal, 
Québec. 


D.R. MILLER, Department of Biological Sciences, National Research Council, 
100 Sussex Drive, Ottawa, Ontario. 


P. ROBERTS-PICHETTE, Executive Secretary, Canadian MAB Programme Secretariat, 
Liaison and Coordination Directorate, Environment Canada, Ottawa, Ontario 
K1A OH3. 


D. SEWELL, Department of Geography, University of Victoria, B.C. 
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